Intelligent anomaly identification and alerting system based on smart ranking of anomalies

ABSTRACT

A method for ranking detected anomalies is disclosed. The method includes generating a graph based on a plurality of rules, wherein the graph comprises nodes representing metrics identified in the rules, edges connecting nodes where metrics associated with connected nodes are identified in a given rule, and edge weights of the edges each representing a severity level assigned to the given rule. The method further includes ranking nodes of the graph based on the edge weights. The method further includes ranking detected anomalies based on the ranking of the nodes corresponding to the metrics associated with the detected anomalies.

CROSS-REFERENCE TO RELATED APPLICATION

The instant Application is a Continuation of, and claims priority to,U.S. Ser. No. 15/152,379 entitled INTELLIGENT ANOMALY IDENTIFICATION ANDALERTING SYSTEM BASED ON SMART RANKING OF ANOMALIES, filed May 11, 2016.The contents of all priority documents are herein incorporated byreference in their entireties.

TECHNICAL FIELD

This disclosure relates in general to the field of computing and, moreparticularly, to intelligent alerting system based on smart ranking ofanomalies.

BACKGROUND

Data centers are common place to computing, especially to cloudcomputing. Data centers generally has many commoditized resources suchas processors, data storages, and even network links. The commoditizedresources can fail or become overloaded. When a resource fails orbecomes overloaded, it is important for an administrator or managementsystem to be notified such that appropriate actions can be taken toensure system up time, maintain quality of service requirements, andreduce further failures. Notifications can often be in a form of logs,metrics, or alerts. Typically, administrators are inundated by thesenotifications, which would normally include a large volume of data,including data reporting status information about resources in the datacenter. When data centers grow larger, so do the volume ofnotifications. It is extremely cumbersome and difficult for a humanadministrator to review all notifications, and management systemssometimes lack the intelligence to understand notifications and reportof anomalies efficiently.

BRIEF DESCRIPTION OF THE DRAWINGS

To provide a more complete understanding of the present disclosure andfeatures and advantages thereof, reference is made to the followingdescription, taken in conjunction with the accompanying figures, whereinlike reference numerals represent like parts, in which:

FIG. 1 shows an exemplary system for alerting, and identifying and/orranking anomalies, according to some embodiments of the disclosure;

FIG. 2 shows an exemplary set of rules for generating alerts;

FIG. 3 shows an exemplary graph generated based on the set of rulesshown in FIG. 2, according to some embodiments of the disclosure;

FIG. 4 is a flow diagram illustrating a method for ranking anomalies,according to some embodiments of the disclosure;

FIG. 5 is a flow diagram illustrating a method for smart thresholding,according to some embodiments of the disclosure; and

FIG. 6 is a flow diagram illustrating a method for combining rules andanomaly detection when triggering an alert, according to someembodiments of the disclosure.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS Overview

A method for ranking detected anomalies is disclosed. The methodincludes generating a graph based on a plurality of rules, wherein thegraph comprises nodes representing metrics identified in the rules,edges connecting nodes where metrics associated with connected nodes areidentified in a given rule (e.g., connected together in the same rulevia one or more logical operations, identified in the same rule), andedge weights of the edges each representing a severity level assigned tothe given rule. The method further includes ranking nodes of the graphbased on the edge weights. The method further includes ranking detectedanomalies based on the ranking of the nodes corresponding to the metricsassociated with the detected anomalies.

In some embodiments, the method further comprises alerting an end userof the detected anomalies based on the ranking of the detectedanomalies.

In some embodiments, the plurality of rules are defined by anadministrator for checking whether certain metrics in the rules meetparticular conditions. In some embodiments, the plurality of rules eachcomprises: at least one metric describing a state of one or moreresources, and at least one condition, wherein each condition is definedfor a corresponding metric.

In some embodiments, the detected anomalies are detected by observingfor anomalous metrics and/or anomalous log through statistical analysis.

In some embodiments, edges are defined in the graph when two or moremetrics and their corresponding conditions are connected in a given rulevia a logical operation. In some embodiments, an edge is defined in thegraph connecting a node to itself when a particular metric is notconnected other metrics in a given rule. In some embodiments, edgeweights are higher in value when a severity level is higher. In someembodiments, value for a given edge weight of a given edge connectingtwo nodes varies depending on a type of logical operation which connectsthe metrics corresponding to the two nodes in a given rule.

In some embodiments ranking nodes comprises: computing an importancevalue for a given node based on all edge weights of edges connected tothe given node, and upon computing importance values for all nodes,ranking the nodes based on the importance values. In some embodiments,ranking nodes comprises: computing an influence value for a given nodeand treating the graph as a social graph, and upon computing influencevalues for all nodes, ranking the nodes based on the influence values.

System for carrying out the method is also disclosed herein.Computer-readable storage medium storing instructions that, whenexecuted by one or more processors, causes the one or more processors tocarry out the method is also disclosed herein.

Example Embodiments

Understanding Data Center, Logs, and Metrics

A data center has many different types of resources (virtual orphysical) operating together to provide services to users and clients.To allow an administrator to monitor the state or health of a datacenter, logs logging activity or events in the data center and metricsrepresenting certain performance or state of resources are reported toan administrator. Since there are many resources in a data center,administrators are often inundated with streams of logs and metrics,making it difficult for a human administrator to monitor and identifyproblems in the data center. When an administrator can not readilyidentify and perform actions to address problems before or soon afterthey occur, services running in the data center can suffer fromlimitations on performance, or worse yet, partial or total systemfailure. To address the issue, systems can implement one of two ways tohelp administrators determine if there are problems in the data center.

One way to make it easier for administrators to understand logs andmetrics is to allow administrators to define rules that triggers alertsto be sent and/or displayed to the administrator. A rule-based alert canbe generated when a metric meets a particular condition (e.g. CPUutilization>90%), and the metric is being monitored in real time. Theserules can be in the form of a predefined threshold (i.e., condition) ona particular metric. An administrator no longer has to manually reviewlogs for problems, and the rules can check for certain conditions in themetrics for the administrator. The workload to be done by theadministrator is greatly reduced. However, this approach can beproblematic in some scenarios. The effectiveness of rule-based alertslargely depends on the domain expertise of the administrator. If rulesare defined narrowly, rules may not capture all potential problems inthe data center. If rules are defined broadly, rules may generate toomany alerts. Given the scale of the data center (or multiple datacenters) being monitored, the administrator can be overwhelmed with thelarge number of alerts at any given instant.

Another way to make it easier for administrators to understand logs andmetrics is the use of anomaly detection. Anomaly detection monitors adata center by identifying (statistical) anomalies in the logs ormetrics. Specifically, anomaly detection automatically identifiesdeviation from normal behavior for the metrics being monitored. Whendeviation or anomalous behavior is identified, an alert is generated foran administrator. Whereas rule-based alerting leverages the domainknowledge and expertise of the administrator who is responsible fordefining those rules, anomaly detection leverages machine learning andstatistical analysis to understand normal behavior and attempt toidentify abnormal patterns in the logs and metrics. However, onedownside of anomaly detection is that anomaly detection based alertingcan potentially lead to many false alarms since they typically do notincorporate domain specific intelligence. False alarms wasteadministrator time and effort in addressing problems that do not exist,or worst yet, false alarms obfuscate actual problems in the data center.

System for Anomaly Ranking

Using either rule-based alerting or anomaly detection can beproblematic. To address some of these issues, the present disclosuredescribes a system which can rank anomaly-related alerts so that anadministrator can prioritize them and react to alerts in a timely andefficient manner. For instance, a mechanism can be implemented tocombine domain specific information with intelligent anomaly detectionalerting systems to rank anomaly-related alerts. Moreover, somemechanisms based on smarter rules can be implemented to reduce noisyalerts.

FIG. 1 shows an exemplary system for alerting, and identifying and/orranking anomalies, according to some embodiments of the disclosure. Thenetwork environment 100 comprises a system 110 for identifying and/orranking anomalies in a data center 120. The system 110 can communicatewith various resources (illustrated as cubes) via a network 130. Eachresource or manager(s) of said resources can be associated with one ormore services 140 implementing a logs and/or metrics collector 142 forproviding real-time logs and/or metrics associated with the resources tothe system 110. The system 110 can also communicate with the one or moreservices 140, via the network 130, or via another network not shown inFIG. 1. The logs and/or metrics collector 142 can provide real-time logsand/or metrics associated with resources in data center 120, and thedata center 120 in part or as a whole, to the system 110. In order togenerate logs and/or metrics related to the data center 120, theservices 140 could be communicatively connected to the data center 120directly, via the network 130, or via another network not shown in FIG.1.

In at least one embodiment, the system 110 can include a graphgeneration module 112, a node ranking module 114, an anomaly rankingmodule 118. In some cases, the system 116 may further include an anomalydetection module 116. In some cases the anomaly detection module 116 isprovided by a different entity. The system 110 can be implemented foralerting based on smarter rules (with or without anomaly rankingfunctionalities), where in such embodiments, the system can include asmarter rules module 120 for implementing improved anomalyidentification and/or improved alerting. The system 110 can also includeat least one processor 122 and at least one memory element 124, alongwith any other suitable hardware to enable its intended functionality.The system 110 may also include an interface (not shown in FIG. 1) toenable communication with a user device 140, which may be operated by auser such as an administrator. As a result of performing functionalitydescribed herein, the system 110 can produce an alert 150. Optionally,in some embodiments, various repositories may be associated with thesystem 110, including, but not limited to, a logs repository 160 forstoring logs, a rules repository 162 for storing rules (e.g., rulesdefined by an administrator, rules defined by smarter rules module 120),and a metrics repository 164 for storing metrics.

Elements of FIG. 1 may be coupled to one another through one or moreinterfaces employing any suitable connections (wired or wireless), whichprovide viable pathways for network communications. Additionally, one ormore of these elements of FIG. 1 may be combined, divided, or removedfrom the architecture based on particular configuration needs. Networkenvironment 100 may include a configuration capable of transmissioncontrol protocol/internet protocol (TCP/IP) communications for thetransmission and/or reception of packets in the network. Networkenvironment 100 may also operate in conjunction with a user datagramprotocol/IP (UDP/IP), any other suitable protocol, or any suitablecombination thereof where appropriate and based on particular needs. Forpurposes of illustrating the techniques of the system 110, it isimportant to understand the activities that may be present in networkenvironment 100. The following foundational information may be viewed asa basis from which the present disclosure may be properly explained.Such information is offered for purposes of explanation only and,accordingly, should not be construed in any way to limit the broad scopeof the present disclosure and its potential applications.

Turning, again, to the infrastructure of FIG. 1, network 130 representsa series of points or nodes of interconnected communication paths forreceiving and transmitting packets of information that propagate throughthe network environment 100. The network 130 offers a communicativeinterface between various parts seen in FIG. 1, and may include any typeor topology of one or more networks such as a local area network (LAN),wireless local area network (WLAN), metropolitan area network (MAN),virtual local area network (VLAN), Intranet, Extranet, wide area network(WAN) such as the Internet, virtual private network (VPN), any otherappropriate network configuration, or any suitable combination thereofthat facilitates communications in the network environment 100. Thenetwork 130 can comprise any number of hardware or software elementscoupled to (and in communication with) each other through acommunications medium. In at least some embodiments, other elements inthe network environment 100 may also communicate via one or morenetworks such as those described with reference to the network 130. Forease of illustration, however, not all elements of FIG. 1 are depictedwith communication lines traversing the network 130 (e.g., logsrepository 160, rules repository 162, metrics repository 164, userdevice 140, etc.). In the network 130, network traffic, which couldinclude packets, frames, signals, cells, datagrams, protocol data units(PDUs), data, etc., can be sent and received according to any suitablecommunication messaging protocols. Suitable communication messagingprotocols can include a multi-layered scheme such as Open SystemsInterconnection (OSI) model, or any derivations or variants thereof(e.g., Transmission Control Protocol/Internet Protocol (TCP/IP), userdatagram protocol/IP (UDP/IP)). A packet is a unit of data forcommunicating information in a network, and can be routed between asource node (e.g., the system 110) and a destination node (e.g., storagenodes 122(1)-122(N)) via the network 130. A packet includes, but is notlimited to, a source network address, a destination network address, anda payload containing the information to be communicated. By way ofexample, these network addresses can be Internet Protocol (IP) addressesin a TCP/IP messaging protocol. Information is generally represented bydata and, as used herein, ‘data’ refers to any type of binary, numeric,voice, video, media, textual, or script data, or any type of source orobject code, or any other suitable information in any appropriate formatthat may be communicated from one point to another in electronic devicesand/or networks.

Services 140 can be configured to provide system metrics of theresources in the data center 120 to the system 110. System metrics caninclude information related to or summarizing current system stateand/or activity including, but not limited to, number of on-going clientoperations, current central processing unit (CPU) utilization, diskusage or load on the storage nodes, available network bandwidth,remaining disk input/output operations per second (10PS), remaining diskbandwidth, etc. In at least one embodiment, these metrics can be pushedto the system 110 by the metrics collectors in real-time. The system 110or services 140 may store the metrics in metrics repository 164, whichmay be internal to the system 110 or external (entirely or in part). Inother embodiments, services 140 may store real-time system metrics inthe metrics repository 164 without accessing the system 110.

In some cases, services 140 can be configured to provide logs loggingevents and activities in data center 120 to system 110. Logs can includeinformation related to events, errors, device drivers, system changes,etc. In at least one embodiment, these logs can be pushed to the system110 by the services 140 in real-time. The system 110 or services 140 maystore the system logs in logs repository 160, which may be internal tothe system 110 or external (entirely or in part). In other embodiments,services 140 may store real-time system logs in the logs repository 160without accessing the system 110.

The system 110 can be implemented by one or more network elements innetwork environment 100. As used herein, the term ‘network element’ ismeant to encompass servers, processors, modules, routers, switches,cable boxes, gateways, bridges, load balancers, firewalls, inlineservice nodes, proxies, or any other suitable device, component,element, or proprietary appliance operable to exchange information in anetwork environment. This network element may include any suitablehardware, software, components, modules, or interfaces that facilitatethe operations thereof. This may be inclusive of appropriate algorithmsand communication protocols that allow for the effective exchange ofdata or information.

In one implementation, the system 110 includes software or instructionsin memory 124 (i.e., non-transitory computer-readable memory medium) toachieve (or to foster) anomaly identification, alerting, and/or anomalyranking processes, as outlined herein. Note that in one example, thesystem 110 can have an internal structure (e.g., processor 122, memoryelement 124, network interface card, etc.) to facilitate some of theoperations described herein. In other embodiments, these activitiescarried out by system 110 may be executed externally to the system 110,or included in some other network element to achieve this intendedfunctionality. Alternatively, the 110 may include this software (orreciprocating software) that can coordinate with other network elementsin order to achieve the operations, as outlined herein. In still otherembodiments, one or several devices may include any suitable algorithms,hardware, software, firmware, components, modules or interfaces thatfacilitate the operations thereof.

In accordance with at least one embodiment of the present disclosure,the network environment 100 can provide improvements to theaforementioned issues associated with rules-based alerting systems oranomaly detection systems. Specific details on the functionalities ofthe various modules in system 110 are explained with FIGS. 2-6. Any oneof the activities described herein can serve to make the alerting systemmore effective, thereby making it easier for an administrator to monitorand react to problems in a data center.

In some embodiments, the system 110 leverages rules that have alreadybeen predefined for generating alerts to rank detected anomalies.Phrased differently, the system 110 can decide which anomaly might bemore important or ought to have higher priority by learning from andinterpreting the rules in an intelligent manner. Specifically, thesystem 110 can rank the anomalies based on the rules (provided by theadministrator or some other entity) represented as a graph andidentifying the most influential nodes. As a result, the domain-specificknowledge and expertise that was “built in” to the rules areextrapolated and applied to ranking detected anomalies, such that anadministrator can easily decide on which anomaly to act on first. Inmany cases, this can prioritize actions when there are multiple metricsshowing anomaly at the same time. Note that anomaly detection generallyidentifies anomalous metrics at any given time. System 110 can rankalerts triggered by these anomalous metrics based on the priority orimportance associated with these metrics. This scheme is unique becausethis technique learns from rules and applies that learning to ranking ofanomalies, and other systems rank anomalies based on the probability ofoccurrence, or underlying system or network topology or hierarchy.

In some other embodiments, system 110 can reduce noise from alerts. Forinstance, anomaly detection systems can change thresholds of rules tomake the rules smarter. In another instance, system 110 can combinerule-based alerting with anomaly detection based alerting methods. Inthis way, system 110 can combine domain knowledge with automatic anomalydetection methods to assist in anomaly identification and/or reducenoise from alerts.

Rules for Generating Alerts

To generate the priorities for the metrics, rules such asthreshold-based rules can be used. Domain-specific expertise andknowledge can be learned from these threshold-based rules. These rulescan be predefined by an administrator. To understand how system 110 canlearn from rules, FIG. 2 shows an exemplary set of rules for generatingalerts. These rules are typically defined by an administrator, but somerules can be generated using the smarter rules module 120 as well. Therules can be stored in the rules repository 162. For simplicity, threethreshold-based rules are shown. Other kinds and examples rules areenvisioned by the disclosure. Generally speaking, rules are defined byan administrator for checking whether certain metrics (e.g., metricsgenerated by a service about a data center 120) in the rules meetparticular conditions. If the metric(s) meet the specified condition(s),the system 110 can generate an alert for the administrator. Such analert may have a predefined severity level or value defined by the rule.Typically, an administrator would typically assign a higher severitylevel if the conditions indicate a more severe or urgent problem in thedata center.

For instance, a rule can include at least one metric describing a stateof one or more resources, and at least one condition, wherein eachcondition is defined for a corresponding metric. A metric and itscorresponding condition is referred herein as a “metric-condition pair”.A rule can have any combination or number of suitable “metric-conditionpairs”. As shown in FIG. 2, Rule 1 checks whether CPU Utilitization“cpu_utilization” (metric) is greater than 90% (condition). Rule 1 alsochecks whether Disk Read Latency “disk_read” (metric) is greater than 50ms (condition). Furthermore, the two metric-condition pairs areconnected in Rule 1 via an “AND” (logical operation). Accordingly, analerting system would check to see if both metric-condition pairs inRule 1 are met before triggering an alert. The alert generated based onRule 1 can have a severity level or value of 10. As shown in FIG. 2,Rule 2 checks whether Queue Length “queue_length” (metric) is greaterthan 100 (condition). Rule 2 also checks whether disk_read (metric) isgreater than 50 ms (condition). Furthermore, the two metric-conditionpairs are connected in Rule 2 via an “AND” (logical operation).Accordingly, an alerting system would check to see if bothmetric-condition pairs in Rule 2 are met before triggering an alert. Thealert generated based on Rule 2 can have a severity level or value of 8.Also shown in FIG. 2, Rule 3 checks whether Disk Write Latency“disk_write” (metric) is greater than 100 ms (condition). Themetric-condition pair is not connected to anything else in Rule 3.Accordingly, an alerting system would check just whether disk_write isgreater than 100 ms before triggering an alert. The alert generatedbased on Rule 3 can have a severity level or value of 5.

Generally speaking, a rule can have any number of metric-condition pairs(e.g., one or more metric-condition pairs). Any suitable combinations ofmetric-condition pairs can be defined in a rule. Furthermore, themetric-connection pairs in a given rule can be connected with each othervia a variety of logical operations, e.g., “AND”, “OR”, “XOR”, etc.While in some cases rules can be written in a different format/syntax,all rules can be written in an equivalent form where metric-conditionpairs are connected by these logical operations.

Generating a Graph Based on the Rules

Using these rules, a graph generation module (e.g., graph generationmodule 112 of FIG. 1) can generate a graph with nodes representing themetric and edges representing the fact that the two metrics areconnected by a rule defined by the administrator. The administratorwhile defining rules also defines the severity level or value of thealert corresponding to the rule. These severity levels or values can becaptured by edge weights of the graph, the higher the severity level ofthe rule the larger/greater is the corresponding edge weight.Advantageously, this graph enables subsequent extraction and applicationof the domain specific knowledge and expertise that an administrator hadwhen he/she defined the rules.

FIG. 3 shows an exemplary graph generated based on the set of rulesshown in FIG. 2, according to some embodiments of the disclosure. Thisgraph corresponds to the three rules shown in FIG. 2. Graphs can begenerated based on other exemplary rules. The three rules in FIG. 2identifies four different metrics: CPU Utilitization “cpu_utilization”,Disk Read Latency “disk_read”, Queue Length “queue_length”, and DiskWrite Latency “disk_write”. Accordingly, the graph has four nodes:cpu_utilization node 302, disk_read node 304, queue_length node 306, anddisk_write node 308. Edges are defined in the graph when two or moremetrics and their corresponding conditions are connected in a given rulevia a logical operation. Since Rule 1 connects [cpu_utilization, greaterthan 90%] and [disk_read, greater than 50 ms] (via an “AND” operation),an edge 310 connects cpu_utilization node 302 and disk_read node 304.Since Rule 2 connects [queue_length, greater than 100] and [disk_read,greater than 50 ms] (via an “AND” operation), an edge 312 connectsqueue_length node 306 and disk_read node 304. Rule 3 only has onemetric-condition pair, and an edge 314 can be defined which originatesfrom disk_write node 308 back to disk_write node 308. If there is onlyone-metric condition pair, the metric-condition pair is consideredconnected to itself. For example, the disk_write node is connected to anode which is itself (hence the edge which points back to the disk_writenode 308).

Graphs are not limited to the example shown. In some cases, rulesconnect more than two metric-condition pairs. For instance, a rule mayinclude [metric-condition-1] AND [metric-condition-2] AND[metric-condition-3]. In such a case, an edge can be defined perpairwise connection, e.g., an edge connecting [metric-condition-1] with[metric-condition-2], an edge connecting [metric-condition-1] with[metric-condition-3], and an edge connecting [metric-condition-2] with[metric-condition-3].

Edge weights corresponding to the edges (e.g., edge 310, 312, and 314)are higher in value when a severity level of the given rule connectingthe metrics associated with the nodes is higher. The edge weight can becorrelated to the severity level of the given rule. In one embodiment,the edge weight is equal to the severity level or value of the rulewhich connected the metrics. Accordingly, edge weight for edge 310 is 10since Rule 1 connecting cpu_utilization and disk_read has a severitylevel or value of 10. Edge weight for edge 312 is 8, since Rule 2connecting queue_length and disk_read has a severity level or value of8. Edge weight for edge 314 is 5 since Rule 1 has a severity level orvalue of 5.

In some embodiments, the value for a given edge weight (or simply “theedge weight”) of a given edge connecting two nodes varies depending on atype of logical operation which connects the metrics corresponding tothe two nodes in a given rule. For instance, the edge weight may behigher if logical operation connecting the metrics is “AND”, as opposedto “OR”. The different edge weight can be computed using amultiplicative or additive factor. The multiplicative or additive factorcan vary depending on the logical operation connecting the two metrics.Such a variation on the edge weight based on the logical operation maytake into account that not all connections or logical operations shouldbe valued the same way. At a high level, the edge weight represents howimportant a particular metric is in relation with another metric, and ifthe logical operation is an “OR”, the relationship may be less importantor weaker than the situation where the logical operation is an “AND”.

Method for Ranking Anomalies

Once the graph is generated, it is possible to extract domain knowledgeand expertise that is embedded in the rules mathematically. To extractthis information, the nodes in the graph are ranked. Once the nodes havea ranking, i.e., once the metrics corresponding to the nodes have aranking, detected anomalies of those metrics can also be ranked.

FIG. 4 is a flow diagram illustrating a method for ranking (detected)anomalies, according to some embodiments of the disclosure. The methodis described with the network environment 100 of FIG. 1.

In task 402, graph generation module (e.g., graph generation module 112)can generate a graph based on a plurality of rules. This scheme isdescribed in relation to FIG. 3. The graph comprises nodes representingmetrics identified in the rules. Furthermore, the graph includes edges,where an edge connects (two) nodes where metrics associated withconnected nodes are identified in a given rule (e.g., in the same rule).The edges signify that the metrics may be connected or related to eachother since the metrics have been used in the same rule. The edgeweights of the edges can each represent a severity level assigned to thegiven rule (e.g., the single rule where the metrics are identified orconnected together). If the rule identifies only one metric (by itself,not connected to other metrics), an edge originates from the nodeassociated with the one metric and points back to the node itself. Theedge weight of such edge represent a severity level assigned to the rulein which the one metric is identified.

In task 404, node ranking module 114 can rank nodes of the graph basedon the edge weights. In one embodiment, to evaluate the rank of each ofthese nodes in the graph, a value is computed based on the sum of edgeweights of all the edges connected to that node. Based on the exampleshown in FIG. 3, a “score” can be computed for each node as follows:

-   -   Rank(disk_read)=10+8=18    -   Rank(cpu_utilization)=10    -   Rank(queue_length)=8    -   Rank(disk_write)=5

In other words, the node ranking module 114 can compute an importancevalue for a given node based on all edge weights of edges connected tothe given node. For instance, the importance value can be a sum of alledge weights. The importance value can be a weighted sum of all edgeweights (if edge weights have a corresponding weight value that can beused, e.g., a weight value which corresponds to the logical operationthat connects the metrics). The importance value can be a product of alledge weights (i.e., computed by multiplying all edge weights together).The importance value can be a mean/mode/median value of all edgeweights.

The node ranking module 114 can, upon computing importance values forall nodes, rank the nodes based on the importance values. For instance,the nodes can be sorted from highest to lowest importance values. Theranking can be higher if the importance value is higher. Based on theexample above, the ranking of the nodes (highest importance value tolowest importance value) is as follows:

-   -   1. disk_read    -   2. cpu_utilization    -   3. queue_length    -   4. disk_write

Other ranking schemes for nodes can be used. For instance, thresholdscan be applied to the importance values to sort the nodes into differentbins having different ranking levels (and thresholds). For instance, theranking of the nodes can be as follows:

-   -   1. SEVERE [metrics having importance value of 10 or more]:        disk_read    -   2. WARNING [metrics having importance value of 6 or more but        less than 10]: cpu_utilization, queue_length    -   3. ALERT [metrics having importance value of less than 6]:        disk_write

It is also possible to apply other schemes to identify most influentialnodes the graph (consider the graph generated from the rules as a socialgraph with weighted edges). The node ranking module 114 can compute aninfluence value for a given node and treat the graph as a social graph,and upon computing influence values for all nodes, the node rankingmodule 114 can rank the nodes based on the influence values.

This ranking of nodes can be used for ranking anomaliesidentified/detected in these metrics corresponding to the nodes. At agiven time, anomalous behavior can be detected in one or more of themetrics mentioned. For example, the multiple alerts that are triggeredby anomalies detected in different metrics can then be sorted or rankedbased on the ranking of the nodes. In task 406, anomaly ranking module118 can rank detected anomalies (e.g., detected by anomaly detectionmodule 116) based on the ranking of the nodes corresponding to themetrics associated with the detected anomalies. For instance, ifanomalous behavior is detected in disk_read and anomalous behavior isdetected in disk_write, the two anomalous behavior (and alerts triggeredtherefrom) can have different ranking. The anomalous behavior fordisk_read can be ranked higher than the anomalous behavior fordisk_write. Generally speaking, the detected anomalies are detected byobserving for anomalous metrics and/or anomalous log through statisticalanalysis.

In task 406, system 110 can alert an end user of the detected anomaliesbased on the ranking of the detected anomalies. The ranking of detectedanomalies can help an administrator decide more easily which detectedanomaly is more important/severe/urgent when more than one anomaly ispredicted/present at a given time. For instance, the anomalies can bedisplayed in an order according to the ranking of the detectedanomalies. The anomalies can be displayed with visual cues whichrepresent the ranking of the detected anomalies. The anomalies can betransmitted differently depending on the ranking of the detectedanomalies (e.g., via different modes of communication). For instance, ifthe ranking of the detected anomaly is high, an administrator may benotified via phone rather than a text message or email. In anotherinstance, if the ranking of the detected anomaly is high, anadministrator may be notified by an audible alarm. In yet anotherinstance, if the ranking of the detected anomaly is high, a differentadministrator may be notified. In yet another instance, if the rankingof the detected anomaly is high, a different action may be triggered.

Smart Thresholding for Rules

FIG. 5 is a flow diagram illustrating a method for smart thresholding,according to some embodiments of the disclosure. The tasks in thismethod can be carried out by smarter rules module 120 of FIG. 1. At task502, the incoming metric data (e.g. CPU_utilization, disk_read,disk_write, queue_length, etc.) is monitored. At task 504, a predictivemodel is applied to the monitored metric data to learn the normalbehavior of the metric. At task 506, based on the output of thepredictive model, a threshold value for a rule for generating alerts canbe modified. Accordingly, smarter rules module 120 can generate dynamicrules for alerts. In task 508, the modified rule can be applied to themetrics to generate/trigger alerts which are smarter and less noisywhen/if the threshold value can be improved.

For instance, suppose the administrator creates a rule-based alert thatstates that if the CPU utilization (metric) goes above 90% (thresholdvalue for the condition) then send an alert. For smart thresholding, thesystem learns how CPU utilization changes over time by using predictivemodels (e.g.: Holt-Winters, ARIMA) on time-series data representing CPUutilization over time. Suppose the utilization reaches 91% every Mondayand is less than 90% rest of the time then smart rules module 120 canuse the output of the predictive model to modify or change the thresholdvalue to 91% instead of 90% on every Monday. As a result, alerts basedon the modified rules can be less noisy (better at identifying andalerting of issues in the data center).

Smarter Rules by Combining Rules and Anomaly Detection

FIG. 6 is a flow diagram illustrating a method for combining rules andanomaly detection when triggering an alert, according to someembodiments of the disclosure. The tasks in this method can be carriedout by smarter rules module 120 of FIG. 1. Such a method can also helpmake alerts less noisy (better at identifying and alerting of issues inthe data center), by making sure alerts are generated when a rule is metand when an anomaly is detected. Phrased differently, the method canreduce noise from alerts triggered by rule-based alerts by combining itwith anomaly detection and vice versa.

In task 602, a rule associated with a metric is determined. This rulecan be predefined by an administrator, or it can be a rule defined bythe smart rules module 120 in a manner described herein. In task 604, ananomaly detection module detects an anomaly in the metric. In task 606,the rule and whether the anomaly is detected is combined in a combinedrule for triggering alerts (“combined rule for alerting”). In task 608,the combined rule for alerting is applied. If the condition specified inthe rule for the metric from task 602 is met and in task 602, theanomaly is detected in the metric, the combined rule triggers an alertfor an administrator.

For instance, suppose the administrator wants to create an alert on CPUutilization. The administrator can do this by applying anomaly detectionbased alerting rule on this metric. At the same time, the administratoralso knows from experience that if the utilization exceeds 90% ingeneral then there is a chance of failure occurring in the system. Thesystem can allow the administrator so specify and apply a new rule thatcombines both the threshold based rule and anomaly detection based alertby combing the two rules with ‘AND’ condition. For example, in this casethe rule would be: If “CPU utilization is greater than 90%” AND “Anomalydetected in CPU utilization” then trigger an alert. Such a rule can bemore robust and less noisy.

Variations and Implementations

Since embodiments of the methods described herein involve alerting, andidentifying anomalies, and/or ranking anomalies in a data center, afunctional entity performing embodiments of these methods is referred toherein as a “system” or “system for alerting, and identifying and/orranking anomalies”. Such a functional entity could be implemented withinany network element or distributed among a plurality of network elementsassociated with a distributed computing system. For example, one or moreof compute servers that may form a networked cluster in the distributedcomputing system to which the computing resources are connected to maybe configured to implement the anomaly identification, alerting, and/oranomaly ranking features.

Within the context of the disclosure, a cloud of host servers generallyincludes host servers communicably connected over a network. A networkused herein represents a series of points, nodes, or network elements ofinterconnected communication paths for receiving and transmittingpackets of information that propagate through a communication system. Anetwork offers communicative interface between sources and/or hosts, andmay be any local area network (LAN), wireless local area network (WLAN),metropolitan area network (MAN), Intranet, Extranet, Internet, WAN,virtual private network (VPN), or any other appropriate architecture orsystem that facilitates communications in a network environmentdepending on the network topology. A network can comprise any number ofhardware or software elements coupled to (and in communication with)each other through a communications medium.

As used herein in this Specification, the term ‘network element’ orparts shown in FIGS. 1 and 4-9, is meant to encompass suitable elementssuch as servers (physical or virtually implemented on physicalhardware), machines (physical or virtually implemented on physicalhardware), end user devices, routers, switches, cable boxes, gateways,bridges, loadbalancers, firewalls, inline service nodes, proxies,processors, modules, or any other suitable device, component, element,proprietary appliance, or object operable to exchange, receive, andtransmit information in a network environment. These network elements orparts shown in FIGS. 1 and 4-9 may include any suitable hardware,software, components, modules, interfaces, or objects that facilitatethe virtual machine placement operations thereof. This may be inclusiveof appropriate algorithms and communication protocols (which can becarried out by one or more processors) that allow for the effectiveexchange of data or information.

In one implementation, the anomaly identification, alerting, and/oranomaly ranking system and the modules therein described herein mayinclude software to achieve (or to foster) the functions discussedherein for carrying out virtual machine placement optimization where thesoftware is executed on one or more processors to carry out thefunctions. This could include the implementation of instances of graphgeneration module, node ranking module, anomaly detection module,anomaly ranking module, and/or any other suitable element that wouldfoster the activities discussed herein. Additionally, each of theseelements can have an internal structure (e.g., one or more suitableprocessors, one or more memory elements, processor 122 of FIG. 1, memory124 of FIG. 1, etc.) to facilitate some of the operations describedherein. In other embodiments, these functions for anomalyidentification, alerting, and/or anomaly ranking may be executedexternally to these elements, or included in some other network elementto achieve the intended functionality. Alternatively, the anomalyidentification, alerting, and/or anomaly ranking system and the modulestherein may include software (or reciprocating software) that cancoordinate with other network elements in order to achieve the functionsdescribed herein. In still other embodiments, one or several devices mayinclude any suitable algorithms, hardware, software, components,modules, interfaces, or objects that facilitate the operations thereof.

In certain example implementations, the anomaly identification,alerting, and/or anomaly ranking functions outlined herein may beimplemented by logic encoded in one or more non-transitory, tangiblemedia (e.g., embedded logic provided in an application specificintegrated circuit [ASIC], digital signal processor [DSP] instructions,software [potentially inclusive of object code and source code] to beexecuted by one or more processors, or other similar machine, etc.). Insome of these instances, one or more memory elements can store data usedfor the operations described herein. This includes the memory elementbeing able to store instructions (e.g., software, code, etc.) that areexecuted to carry out the activities described in this Specification.The memory element is further configured to store logs, rules, metrics,etc. The processor can execute any type of instructions associated withthe data to achieve the operations detailed herein in thisSpecification. In one example, the processor could transform an elementor an article (e.g., data) from one state or thing to another state orthing. In another example, the activities outlined herein may beimplemented with fixed logic or programmable logic (e.g.,software/computer instructions executed by the processor) and theelements identified herein could be some type of a programmableprocessor, programmable digital logic (e.g., a field programmable gatearray [FPGA], an erasable programmable read only memory (EPROM), anelectrically erasable programmable ROM (EEPROM)) or an ASIC thatincludes digital logic, software, code, electronic instructions, or anysuitable combination thereof.

As will be appreciated by one skilled in the art, aspects of the presentdisclosure, in particular the functionality of the anomalyidentification, alerting, and/or anomaly ranking system describedherein, may be embodied as a system, a method or a computer programproduct. Accordingly, aspects of the present disclosure may take theform of an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Functions described in this disclosure may be implemented as analgorithm executed by a processor, e.g. a microprocessor, of a computer.Furthermore, aspects of the present disclosure may take the form of acomputer program product embodied in one or more computer readablemedium(s), preferably non-transitory, having computer readable programcode embodied, e.g., stored, thereon. In various embodiments, such acomputer program may, for example, be downloaded to the existing devicesand systems (e.g. to the existing network elements such as the existingservers, routers, switches, various control nodes, etc.) or be storedupon manufacturing of these devices and systems.

Any of these elements (e.g., the network elements, etc.) can includememory elements for storing information to be used in achieving anomalyidentification, alerting, and/or ranking functions, as outlined herein.Additionally, each of these devices may include a processor that canexecute software or an algorithm to perform the activities as discussedin this Specification. These devices may further keep information in anysuitable memory element [random access memory (RAM), ROM, EPROM, EEPROM,ASIC, etc.], software, hardware, or in any other suitable component,device, element, or object where appropriate and based on particularneeds. Any of the memory items discussed herein should be construed asbeing encompassed within the broad term ‘memory element.’ Similarly, anyof the potential processing elements, modules, and machines described inthis Specification should be construed as being encompassed within thebroad term ‘processor.’ Each of the network elements can also includesuitable interfaces for receiving, transmitting, and/or otherwisecommunicating data or information in a network environment.

Additionally, it should be noted that with the examples provided above,interaction may be described in terms of two, three, or four networkelements or parts shown in FIGS. 1. However, this has been done forpurposes of clarity and example only. In certain cases, it may be easierto describe one or more of the functionalities of a given set of flowsby only referencing a limited number of network elements. It should beappreciated that the systems described herein are readily scalable and,further, can accommodate a large number of components, as well as morecomplicated/sophisticated arrangements and configurations. Accordingly,the examples provided should not limit the scope or inhibit the broadtechniques of anomaly identification, alerting, and/or anomaly ranking,as potentially applied to a myriad of other architectures.

It should also be noted that many of the previous discussions may implya single client-server relationship. In reality, there is a multitude ofservers in the delivery tier in certain implementations of the presentdisclosure. Moreover, the present disclosure can readily be extended toapply to intervening servers further upstream in the architecture,though this is not necessarily correlated to the ‘m’ clients that arepassing through the ‘n’ servers. Any such permutations, scaling, andconfigurations are clearly within the broad scope of the presentdisclosure.

It is also important to note that the steps in the FIG. 4-6 illustrateonly some of the possible scenarios that may be executed by, or within,the anomaly identification, alerting, and/or anomaly ranking systemdescribed herein. Some of these steps may be deleted or removed whereappropriate, or these steps may be modified or changed considerablywithout departing from the scope of the present disclosure. In addition,a number of these operations have been described as being executedconcurrently with, or in parallel to, one or more additional operations.However, the timing of these operations may be altered considerably. Thepreceding operational flows have been offered for purposes of exampleand discussion. Substantial flexibility is provided by anomalyidentification, alerting, and/or anomaly ranking system in that anysuitable arrangements, chronologies, configurations, and timingmechanisms may be provided without departing from the teachings of thepresent disclosure.

Numerous other changes, substitutions, variations, alterations, andmodifications may be ascertained to one skilled in the art and it isintended that the present disclosure encompass all such changes,substitutions, variations, alterations, and modifications as fallingwithin the scope of the appended claims. In order to assist the UnitedStates Patent and Trademark Office (USPTO) and, additionally, anyreaders of any patent issued on this application in interpreting theclaims appended hereto, Applicant wishes to note that the Applicant: (a)does not intend any of the appended claims to invoke paragraph six (6)of 35 U.S.C. section 112 as it exists on the date of the filing hereofunless the words “means for” or “step for” are specifically used in theparticular claims; and (b) does not intend, by any statement in thespecification, to limit this disclosure in any way that is not otherwisereflected in the appended claims.

Although the claims are presented in single dependency format in thestyle used before the USPTO, it should be understood that any claim candepend on and be combined with any preceding claim of the same typeunless that is clearly technically infeasible.

What is claimed is:
 1. A method for ranking detected anomalies, themethod comprising: defining a plurality of rules; generating a graph ofthe plurality of rules, comprising: creating a plurality of nodes, eachof the nodes corresponding to a metric of the plurality of rules, therules including a first rule in which a single metric is a potentialsource of an anomaly and a second rule in which a pair of metrics is apotential source of anomaly; connecting, for each node corresponding toa single metric of the rules, the corresponding each node to itself withan edge; connecting, for each pair of metrics of the second rules, thecorresponding pair of nodes with edges; assigning each of the edges withan edge weight corresponding to a severity level of the correspondingrule that defined the edge; detecting anomalies by comparing systemmetrics relative to the rules; and ranking detected anomalies based onthe ranking of the nodes corresponding to the metrics of the graph;wherein the defining occurs before the detecting.
 2. The method of claim1, further comprising: alerting an end user of the detected anomaliesbased on the ranking of the detected anomalies.
 3. The method of claim1, wherein the plurality of rules each comprises: at least one metricdescribing a state of one or more resources; and at least one condition,wherein each condition is defined for a corresponding metric.
 4. Themethod of claim 1, wherein the detected anomalies are detected byobserving for anomalous metrics and/or anomalous log through statisticalanalysis.
 5. The method of claim 1, wherein edges are defined in thegraph when two or more metrics and their corresponding conditions areconnected in a given rule via a logical operation.
 6. The method ofclaim 1, wherein value for a given edge weight of a given edgeconnecting two nodes varies depending on a type of logical operationwhich connects metrics corresponding to the two nodes.
 7. The method ofclaim 1, wherein ranking nodes comprises: computing an importance valuefor a given node based on all edge weights of edges connected to thegiven node; and upon computing importance values for all nodes, rankingthe nodes based on the importance values.
 8. A non-transitorycomputer-readable medium comprising one or more instructions that whenexecuted on a processor cause the processor to perform operationscomprising: define a plurality of rules; generate a graph of theplurality of rules, comprising: create a plurality of nodes, each of thenodes corresponding to a metric of the plurality of rules, the rulesincluding a first rule in which a single metric is a potential source ofan anomaly and a second rule in which a pair of metrics is a potentialsource of anomaly; connect, for each node corresponding to a singlemetric of the rules, the corresponding each node to itself with an edge;connect, for each pair of metrics of the second rules, the correspondingpair of nodes with edges; assign each of the edges with an edge weightcorresponding to a severity level of the corresponding rule that definedthe edge; detect anomalies by comparing system metrics relative to therules; and rank detected anomalies based on the ranking of the nodescorresponding to the metrics of the graph; wherein the define occursbefore the detect.
 9. The non-transitory computer-readable medium ofclaim 8, the operations further comprising: alerting an end user of thedetected anomalies based on the ranking of the detected anomalies. 10.The non-transitory computer-readable medium of claim 8, wherein theplurality of rules each comprises: at least one metric describing astate of one or more resources; and at least one condition, wherein eachcondition is defined for a corresponding metric.
 11. The non-transitorycomputer-readable medium of claim 8, wherein the detected anomalies aredetected by observing for anomalous metrics and/or anomalous log throughstatistical analysis.
 12. The non-transitory computer-readable medium ofclaim 8, wherein edges are defined in the graph when two or more metricsand their corresponding conditions are connected in a given rule via alogical operation.
 13. The non-transitory computer-readable medium ofclaim 8, wherein value for a given edge weight of a given edgeconnecting two nodes varies depending on a type of logical operationwhich connects metrics corresponding to the two nodes.
 14. Thenon-transitory computer-readable medium of claim 8, wherein theoperation of rank nodes comprises: compute an importance value for agiven node based on all edge weights of edges connected to the givennode; and upon compute importance values for all nodes, rank the nodesbased on the importance values.
 15. A system, comprising: anon-transitory computer readable media storing instructions; a processorprogrammed to cooperate with the instructions to perform operationscomprising: define a plurality of rules; generate a graph of theplurality of rules, comprising: create a plurality of nodes, each of thenodes corresponding to a metric of the plurality of rules, the rulesincluding a first rule in which a single metric is a potential source ofan anomaly and a second rule in which a pair of metrics is a potentialsource of anomaly; connect, for each node corresponding to a singlemetric of the rules, the corresponding each node to itself with an edge;connect, for each pair of metrics of the second rules, the correspondingpair of nodes with edges; assign each of the edges with an edge weightcorresponding to a severity level of the corresponding rule that definedthe edge; detect anomalies by comparing system metrics relative to therules; and rank detected anomalies based on the ranking of the nodescorresponding to the metrics of the graph; wherein the define occursbefore the detect.
 16. The system of claim 15, the operations furthercomprising: alerting an end user of the detected anomalies based on theranking of the detected anomalies.
 17. The system of claim 15, whereinthe plurality of rules each comprises: at least one metric describing astate of one or more resources; and at least one condition, wherein eachcondition is defined for a corresponding metric.
 18. The system of claim15, wherein the detected anomalies are detected by observing foranomalous metrics and/or anomalous log through statistical analysis. 19.The system of claim 15, wherein edges are defined in the graph when twoor more metrics and their corresponding conditions are connected in agiven rule via a logical operation.
 20. The system of claim 15, whereinvalue for a given edge weight of a given edge connecting two nodesvaries depending on a type of logical operation which connects metricscorresponding to the two nodes.